{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "DSDT_Keras.ipynb",
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/nrkfeller/YCBS258/blob/master/DSDT_Keras.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xhW0sbXbdnC0",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "!pip install -q pyyaml"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0B3_fj18q-Sz",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Install dependencies\n",
        "!apt install graphviz\n",
        "!pip install pydot pydot-ng\n",
        "!echo \"Double check with Python 3\"\n",
        "!python -c \"import pydot\""
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7zwaBdZJhdPZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import Sequential, layers\n",
        "from tensorflow.keras.layers import Dense, Flatten\n",
        "\n",
        "print(tf.VERSION)\n",
        "print(tf.keras.__version__)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oPCGxnvKKMgt",
        "colab_type": "text"
      },
      "source": [
        "### Models types\n",
        "### Sequential\n",
        "Simple stack of layers\n",
        "### Functional\n",
        "Multi input, multi output, shared layers, non sequential flows\n",
        "### Model Subclassing\n",
        "Customizable, define foward pass with ```call``` method. Enables [eager execution](https://www.tensorflow.org/guide/eager#build_a_model)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T3gDzVSZM3DX",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import numpy as np\n",
        "import pandas as pd\n",
        "\n",
        "features = np.random.random((1000, 32))\n",
        "labels = np.random.random((1000, 10))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "W65lRhammQkO",
        "colab_type": "text"
      },
      "source": [
        "### Sequential Model"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2jJXz3MkhlsB",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# maximum simplicity\n",
        "seq_model = Sequential()\n",
        "seq_model.add(Dense(20, activation='relu', input_shape=(32,)))\n",
        "seq_model.add(Dense(20, activation='relu'))\n",
        "seq_model.add(Dense(10, activation='softmax'))\n",
        "\n",
        "seq_model.compile(optimizer=tf.train.AdamOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "\n",
        "seq_model.fit(features, labels, epochs=10, batch_size=32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "KQhkaZ28mTA9",
        "colab_type": "text"
      },
      "source": [
        "### Functional Model\n",
        "[example](https://cdn-images-1.medium.com/max/2600/1*6hF97Upuqg_LdsqWY6n_wg.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VBHDpI2kKY7b",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# more full featured and flexible\n",
        "inputs = keras.Input(shape=(32,))\n",
        "x = Dense(20, activation='relu')(inputs)\n",
        "x = Dense(20, activation='relu')(x)\n",
        "outputs = Dense(10, activation='softmax')(x)\n",
        "\n",
        "func_model = keras.Model(inputs, outputs)\n",
        "func_model.compile(optimizer=tf.train.AdamOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "func_model.fit(features, labels, epochs=10, batch_size=32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "YRa8kh2-mVGf",
        "colab_type": "text"
      },
      "source": [
        "### Model Subclassing"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PA-jfeUzLdKO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# maximally flexible and hackable\n",
        "class ScModel(keras.Model):\n",
        "  \n",
        "  def __init__(self):\n",
        "    super(ScModel, self).__init__()\n",
        "    self.dense1 = Dense(20, activation='relu')\n",
        "    self.dense2 = Dense(20, activation='relu')\n",
        "    self.dense3 = Dense(10, activation='softmax')\n",
        "    \n",
        "  def call(self, inputs):\n",
        "    x = self.dense1(inputs)\n",
        "    x = self.dense2(x)\n",
        "    return self.dense3(x)\n",
        "  \n",
        "sc_model = ScModel()\n",
        "sc_model.compile(optimizer=tf.train.AdamOptimizer(0.001),\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "sc_model.fit(features, labels, epochs=10, batch_size=32)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "j3ItCImGoDSg",
        "colab_type": "text"
      },
      "source": [
        "### Visualize model\n",
        "\n",
        "The summary is useful for simple models, but can be confusing for models that have multiple inputs or outputs.\n",
        "\n",
        "Keras also provides a function to create a plot of the network neural network graph that can make more complex models easier to understand.\n",
        "\n",
        "The plot_model() function in Keras will create a plot of your network. This function takes a few useful arguments:\n",
        "\n",
        "* ```model```: (required) The model that you wish to plot.\n",
        "* ```to_file```: (required) The name of the file to which to save the plot.\n",
        "* ```show_shapes```: (optional, defaults to False) Whether or not to show the output shapes of each layer.\n",
        "* ```show_layer_names```: (optional, defaults to True) Whether or not to show the name for each layer."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7KmC42hroBol",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sc_model.summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "q7cDLp2yp3kY",
        "colab_type": "text"
      },
      "source": [
        "[fully connected NN](https://www.oreilly.com/library/view/tensorflow-for-deep/9781491980446/assets/tfdl_0402.png)"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Wrzmlp4npIXK",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "mod = func_model\n",
        "\n",
        "tf.keras.utils.plot_model(\n",
        "    mod, \n",
        "    to_file='{}.png'.format(mod), \n",
        "    show_shapes=True, \n",
        "    show_layer_names=True\n",
        ")\n",
        "\n",
        "from IPython.display import Image\n",
        "Image(retina=True, filename='{}.png'.format(mod))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "90NHWXygtgu0",
        "colab_type": "text"
      },
      "source": [
        "### Evaluate and predict\n",
        "\n",
        "The `tf.keras.Model.evaluate` and `tf.keras.Model.predict` methods can use NumPy\n",
        "data and a `tf.data.Dataset`.\n",
        "\n",
        "To *evaluate* the inference-mode loss and metrics for the data provided:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fW5GZmB_jI_w",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Unseen data\n",
        "new_data  = np.random.random((1000, 32))\n",
        "new_labels = np.random.random((1000, 10))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "TnbrMxQMjjN2",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "results = sc_model.predict(new_data)\n",
        "results[0]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "H6tOtNLwjpje",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "sc_model.evaluate(new_data, new_labels)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IwzvqjrLlJrZ",
        "colab_type": "text"
      },
      "source": [
        "## Callbacks\n",
        "\n",
        "A callback is an object passed to a model to customize and extend its behavior\n",
        "during training. You can write your own custom callback, or use the built-in\n",
        "`tf.keras.callbacks` that include:\n",
        "\n",
        "* `tf.keras.callbacks.ModelCheckpoint`: Save checkpoints of your model at\n",
        "  regular intervals.\n",
        "* `tf.keras.callbacks.LearningRateScheduler`: Dynamically change the learning\n",
        "  rate.\n",
        "* `tf.keras.callbacks.EarlyStopping`: Interrupt training when validation\n",
        "  performance has stopped improving.\n",
        "* `tf.keras.callbacks.TensorBoard`: Monitor the model's behavior using\n",
        "  [TensorBoard](./summaries_and_tensorboard.md).\n",
        "\n",
        "To use a `tf.keras.callbacks.Callback`, pass it to the model's `fit` method:"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "SkXcbKEflI7W",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "callbacks = [\n",
        "  tf.keras.callbacks.EarlyStopping(patience=2, monitor='val_loss'),\n",
        "  tf.keras.callbacks.TensorBoard(log_dir='./logs')\n",
        "]\n",
        "model.fit(features, labels, batch_size=32, epochs=5, callbacks=callbacks,\n",
        "          validation_data=(new_data, new_labels))"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "31yq6mUxlZxN",
        "colab_type": "text"
      },
      "source": [
        "## Model Saving and Restoring\n",
        "### Weights only\n",
        "\n",
        "Save and load the weights of a model using `tf.keras.Model.save_weights`:\n",
        "\n",
        "### Configuration only\n",
        "\n",
        "A model's configuration can be saved—this serializes the model architecture\n",
        "without any weights. A saved configuration can recreate and initialize the same\n",
        "model, even without the code that defined the original model. Keras supports\n",
        "JSON and YAML serialization formats:\n",
        "\n",
        "### Entire model\n",
        "\n",
        "The entire model can be saved to a file that contains the weight values, the\n",
        "model's configuration, and even the optimizer's configuration. This allows you\n",
        "to checkpoint a model and resume training later—from the exact same\n",
        "state—without access to the original code."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "af1xTnLMlUwS",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "model = tf.keras.Sequential([\n",
        "  layers.Dense(10, activation='softmax', input_shape=(32,)),\n",
        "  layers.Dense(10, activation='softmax')\n",
        "])\n",
        "\n",
        "model.compile(optimizer='rmsprop',\n",
        "              loss='categorical_crossentropy',\n",
        "              metrics=['accuracy'])\n",
        "model.fit(features, labels, batch_size=32, epochs=5)\n",
        "\n",
        "model.save('my_model.h5')\n",
        "\n",
        "model = tf.keras.models.load_model('my_model.h5')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dUrsPbBabv1x",
        "colab_type": "text"
      },
      "source": [
        "## Futher Reading\n",
        "* https://github.com/Dataweekends/zero_to_deep_learning_video \n",
        "* https://github.com/keras-team/keras/tree/master/examples \n",
        "* https://github.com/PacktPublishing/Advanced-Deep-Learning-with-Keras \n",
        "* https://medium.com/tensorflow/tagged/keras\n",
        "* https://github.com/keras-team/keras"
      ]
    }
  ]
}